Syntactic Vs. Semantic Similarity of Artificial and Real Faults in Mutation Testing Studies

Fault seeding is typically used in empirical studies to evaluate and compare test techniques. Central these techniques lies the hypothesis that artificially seeded faults involve some form of realistic properties thus provide experimental results. In an attempt strengthen realism, a recent line research uses machine learning techniques, such as deep Natural Language Processing, seed look like (syntactically) real ones, implying fault realism related syntactic similarity. This raises question whether syntactically similar indeed results semantically and, more generally dissimilar are far away (semantically) from ones. We answer this by employing 4 state-of-the-art fault-seeding (PiTest - popular mutation testing tool, IBIR tool with manually crafted patterns, DeepMutation learning-based framework μBERT based on pre-trained language model CodeBERT) operate fundamentally different way, demonstrate similarity does not reflect semantic also show 65.11%, 76.44%, 61.39% 9.76% Defects4J V2 resembled PiTest, IBIR, faults, respectively.